Efficient Test-Time Adaptation through Latent Subspace Coefficients Search

Xinyu Luo; Jie Liu; Kecheng Chen; Yihua Shao; Junyi Yang; Bo Ding; Nicu Sebe; Arindam Basu

cs.LGOct 13, 2025

Efficient Test-Time Adaptation through Latent Subspace Coefficients Search

Authors: Xinyu Luo, Jie Liu, Kecheng Chen, Yihua Shao, Junyi Yang, Bo Ding, Nicu Sebe, Arindam Basu,

Organizations: Department of Electrical Engineering, City University of Hong Kong, Hong Kong SAR, China

Abstract

Real-world deployment often exposes vision models to distribution shifts, making test-time adaptation (TTA) important for reliable prediction. Existing TTA methods can be difficult to deploy on edge devices because they require backpropagation, activation storage, or test-time batches, which increase computation, memory, and state. We propose ELaTTA (Efficient Latent Test-Time Adaptation), a forward-only, gradient-free framework for single-instance TTA. ELaTTA freezes the pretrained model and adapts each test input by searching a low-dimensional coefficient vector in a principal latent subspace computed offline from a source-domain cache. The basis is stored once and source data are discarded before deployment. At test time, CMA-ES performs covariance-adaptive Gaussian search, ranks candidate corrections by prediction entropy, and selects the best candidate within a fixed evaluation budget. On five visual benchmarks spanning corruption, natural, rendition, sketch, and cross-domain shifts, ELaTTA provides strong accuracy under strict and continual single-instance protocols. In the strict single-instance comparison, MEMO requires \emph{64.7 $\times$ } the computation and \emph{12.4 $\times$ } the peak memory of ELaTTA. A complementary keyword-spotting experiment shows that the same coefficient-search design transfers beyond vision. We further demonstrate ViT-Base adaptation on a Raspberry Pi 5, confirming the feasibility of deploying ELaTTA on an edge device.

Figures & tables

Figure 1: Accuracy, computation, and peak memory of various TTA methods under the single-instance setting on ImageNet-C with ViT-Base model.

Figure 2: Overview of ELaTTA framework. During offline preparation, we first compute a $k$ -dimensional latent PC basis $\mathbf{V}_{k}$ from the source-domain latents. At test time, for a given single OOD sample, we perform a forward-only adaptation by searching for a low-dimensional coefficient $\mathbf{p}\in\mathbb{R}^{k}$ within the source-induced subspace $\mathrm{span}(\mathbf{V}_{k})$ and then selecting the update that minimizes prediction entropy. We use CMA-ES for this gradient-free optimization, enabling efficient test-time adaptation without costly backpropagation on edge devices.

Model	ViT-Base	ResNet-50	EfficientNet-B0	MobileNet-V4	LSTM
$D$	768	2048	1280	1280	32

Table 1: Five backbone models and their hidden size

D

(first dimension of the latent PC basis

\mathbf{V}_{k}

Method	GF	Noise			Blur				Weather				Digital				Average
Method	GF	Gauss.	Shot	Impl.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Acc.
No Adapt	✓	55.34	56.23	56.01	46.48	34.78	52.87	44.20	62.39	62.66	65.56	77.70	32.04	45.73	66.72	66.67	55.03
FOA	✓	53.87	54.16	54.00	46.17	33.45	52.56	43.69	61.82	62.30	66.17	77.73	30.60	46.14	66.18	66.77	54.37
CAZO	✓	54.24	56.10	55.95	47.13	36.08	53.55	45.26	62.99	61.97	66.00	77.24	33.08	47.25	67.16	67.03	55.40
T3A	✓	54.69	55.95	55.61	47.41	36.77	53.91	46.44	63.85	60.42	68.12	78.11	37.79	49.54	67.24	68.04	56.26
CoTTA	✗	54.61	55.66	55.37	45.28	34.35	52.69	44.11	62.38	62.62	58.33	77.71	29.58	45.65	66.68	66.66	54.11

Table 2: Accuracy (%) on ImageNet-C with ViT-Base under strict single-instance adaptation. GF denotes gradient-free adaptation. Bold marks the best value in each column.

Method	GF	Accuracy (%)
Method	GF	V2	R	Sketch	Avg.
No Adapt	✓	75.49	59.49	44.89	59.96
FOA	✓	75.25	59.96	44.95	60.05
CAZO	✓	75.43	59.78	45.91	60.37
T3A	✓	75.61	57.98	48.44	60.68
CoTTA	✗	75.50	59.20	44.77	59.82

Table 3: Accuracy (%) on ImageNet-V2/R/Sketch with ViT-Base under strict single-instance adaptation. GF denotes gradient-free adaptation. Bold marks the best value in each column.

Method	GF	GFLOPs	Mem (MB)	Latency (s)
FOA	✓	479.31	702	0.273
CAZO	✓	691.47	767	0.382
T3A	✓	16.86	718	0.124
CoTTA	✗	50.59	1130	0.703
SAR	✗	33.73	2996	0.037
PASLE	✗	607.13	2588	0.051

Table 4: Computational comparison (GFLOPs, Memory, and Latency) on ImageNet-C with ViT-Base.

Method	Accuracy (%)	Encoder latency (ms)	Search latency (ms)	End-to-end latency (ms)	Peak process RSS (MB)
No Adapt	55.02	652	N/A	663	1081
ELaTTA ( $k=16$ , $n=8$ )	57.94	652	87	754	1109

Table 5: ViT-Base deployment on Raspberry Pi 5 (8 GB) using PyTorch CPU in FP32 with batch size one. Accuracy is measured on ImageNet-C at severity 5. The encoder runs once per image for both methods. ELaTTA search latency includes latent updates, classifier/entropy evaluations, and CMA-ES sampling and state updates; search is not applicable to No Adapt. Peak process RSS denotes the maximum resident set size of the inference process.

Method	Time $\xrightarrow{\hskip 540.60236pt}$
	Noise			Blur				Weather				Digital				Average
	Gauss.	Shot	Impl.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Acc.
No Adapt	55.34	56.23	56.01	46.48	34.78	52.87	44.20	62.39	62.66	65.56	77.70	32.04	45.73	66.72	66.67	55.03
ZOA	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.06
ROID	0.10	0.08	0.08	0.08	0.08	0.08	0.08	0.08	0.08	0.08	0.08	0.08	0.08	0.08	0.08	0.08
CoTTA	0.13	0.12	0.16	0.12	0.13	0.16	0.13	0.16	0.14	0.17	0.21	0.11	0.15	0.17	0.15	0.15

Table 6: Accuracy (%) on ImageNet-C with ViT-Base under continual single-instance adaptation. Corruptions follow the displayed stream order. Bold marks column maxima.

Method	Time $\xrightarrow{\hskip 540.60236pt}$																Memory (MB)
	Noise			Blur				Weather				Digital				Average
	Gauss.	Shot	Impl.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Acc.
No Adapt	55.34	56.23	56.01	46.48	34.78	52.87	44.20	62.39	62.66	65.56	77.70	32.04	45.73	66.72	66.67	55.03	–
ZOA	58.00	58.56	59.38	47.98	39.10	56.38	49.38	65.68	63.48	62.68	79.34	49.02	50.76	66.84	70.34	58.46	1734
ROID	60.68	63.04	63.00	56.60	55.84	62.72	58.86	66.78	66.70	70.76	79.24	12.14	57.02	69.46	71.04	60.93	10866
FOA	57.14	61.62	61.96	51.98	42.78	57.48	54.08	65.98	68.44	65.60	79.48	62.78	52.20	70.32	72.20	61.60	1300

Table 7: Accuracy (%) and memory use (MB) on ImageNet-C with ViT-Base under continual-batch adaptation (batch size 64). Bold marks the highest accuracies and the lowest reported memory.

Networks	Method	Noise			Blur				Weather				Digital				Average
Networks	Method	Gauss.	Shot	Impl.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Acc.
ResNet-50	No Adapt	4.47	4.74	4.06	8.11	5.96	9.59	14.74	6.92	14.47	12.32	45.80	0.72	11.08	18.19	32.54	12.91
	IABN	7.22	5.10	5.54	5.46	7.38	11.92	15.98	5.74	12.32	14.14	47.10	0.90	7.10	18.26	18.04	12.15
	TTN	1.73	5.13	4.90	4.17	6.69	10.16	14.14	6.13	14.52	14.42	49.20	0.22	12.26	22.03	33.07	13.25
	SAR	2.34	4.46	4.24	6.56	6.66	14.04	16.34	4.02	14.68	13.26	48.51	0.35	12.43	20.44	36.46	13.65
	ELaTTA	4.99	5.18	4.41	11.36	7.32	14.26	18.29	7.00	15.32	15.01	55.32	0.25	13.07	23.29	40.97	15.74

Table 8: Accuracy (%) on ImageNet-C across visual backbones under strict single-instance adaptation. Bold marks the best value for each backbone and corruption.

Method	(a) Painting as Source				(b) Clipart as Source				(c) Real as Source				(d) Sketch as Source
Method	Real	Sketch	Clipart	Avg.	Sketch	Real	Painting	Avg.	Clipart	Painting	Sketch	Avg.	Painting	Clipart	Real	Avg.
No Adapt	74.84	49.70	53.26	59.27	46.16	60.50	43.84	50.17	55.62	61.88	47.84	55.11	48.48	54.88	59.32	54.23
SAR (BS=64)	73.58	53.96	53.50	60.35	48.80	64.36	47.34	53.50	54.36	62.90	48.40	55.22	57.56	58.98	67.92	61.49
ROID (BS=64)	75.04	57.30	57.24	63.19	51.24	64.64	49.14	55.01	57.48	65.18	52.66	58.44	58.76	61.44	68.58	62.93
SAR (BS=1)	6.42	4.24	3.22	4.63	4.98	5.80	4.38	5.05	3.74	4.66	3.58	3.99	4.78	4.50	5.26	4.85
ROID (BS=1)	0.48	0.16	0.16	0.27	0.16	0.48	0.30	0.31	0.16	0.30	0.16	0.21	0.30	0.16	0.48	0.31

Table 10: Accuracy (%) on DomainNet-126 with ResNet-50. Each group specifies the source domain and its three target domains. BS denotes test-time batch size.

SNR	Method	GF	Animals		Natural		Human		Domestic		Urban		Average
SNR	Method	GF	dog	cat	pouring water	thunderstorm	crying baby	laughing	washing machine	vacuum cleaner	car horn	fireworks	Acc.
-10 dB	No Adapt	✓	62.67	61.17	54.55	66.23	58.74	58.59	52.88	50.43	56.62	61.54	58.34
	T3A	✓	62.67	61.17	54.55	66.23	58.74	58.59	52.88	50.43	56.62	61.54	58.34
	PASLE	✗	62.87	62.06	54.78	66.75	59.42	59.63	57.69	53.38	58.35	62.73	59.77
	ELaTTA	✓	64.25	63.58	59.73	66.47	61.99	61.94	59.46	56.98	59.32	64.90	61.86
-15 dB	No Adapt	✓	57.08	53.63	49.35	61.45	53.08	53.03	46.81	47.49	51.76	55.19	52.89

Table 11: Performance comparison on GSC-C with LSTM model regarding Accuracy (%). GF stands for gradient-free. The bold number indicates the best result.

Figure 3: Effect of the subspace dimension

k

and the number of search iterations

n

on ImageNet-C with ViT-Base under strict single-instance adaptation. (a) Average accuracy; the dashed line is No Adapt. (b) Alignment between the descent directions of the search objective and supervised cross-entropy. The circle marks the default setting.

Method	Acc $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
No Adapt	55.03	8.64	2.32
Backprop.	55.52	6.72	2.34
ELaTTA	57.82	6.23	2.03

Table 13: Accuracy and calibration metrics on ImageNet-C with ViT-Base under strict single-instance adaptation. Accuracy and ECE are in %.

Protocol	Accuracy (%)
Strict single-instance	57.86 $\pm$ 0.13
Continual single-instance	59.75 $\pm$ 0.11

Table 16: Variation across search seeds on ImageNet-C with ViT-Base. The source cache is fixed, and each entry is the mean

\pm

standard deviation of the average accuracy over three independent search seeds.

Metric	Value
Correct $\rightarrow$ Correct	27,224
Incorrect $\rightarrow$ Incorrect	20,169
Beneficial Switch	2,161 (4.32%)
Harmful Switch	446 (0.89%)
Correctness Transition Rate	5.21%
Net Gain	+1,715

Table 17: Prediction changes before and after ELaTTA on ImageNet-C Gaussian noise domain with ViT-Base model.

Figure 4: Effect of corruption severity on the Contrast corruption of ImageNet-C under strict single-instance adaptation. (a) Accuracy of No Adapt (dashed) and ELaTTA (solid). (b) Accuracy change of ELaTTA relative to No Adapt. Severities 1, 3, and 5 were evaluated.

Method	Noise			Blur				Weather				Digital				Average
Method	Gauss.	Shot	Impl.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Acc.
ELaTTA	58.77	59.66	59.50	49.30	36.08	55.35	46.34	65.21	66.40	67.66	80.21	35.96	47.61	69.55	69.68	57.82
QELaTTA-V1	59.41	60.15	60.09	49.86	36.48	55.94	46.70	65.60	66.74	60.90	80.46	34.55	47.72	69.83	70.01	57.63
QELaTTA-V2 (8b5)	56.16	57.11	56.93	47.17	35.14	53.60	44.76	63.18	63.94	67.60	78.37	33.03	46.26	67.48	67.49	55.88
QELaTTA-V2 (8b4)	56.75	57.75	57.35	47.63	35.32	53.99	45.12	63.60	64.39	68.34	78.66	33.71	46.51	67.83	67.88	56.32
QELaTTA-V2 (8b3)	56.73	57.61	57.42	47.63	35.35	53.90	45.04	63.49	64.35	68.07	78.65	33.16	46.53	67.75	67.77	56.23

Table 18: Accuracy (%) of ELaTTA and its quantized variants on ImageNet-C at severity 5 with ViT-Base under strict single-instance adaptation. QELaTTA-V2 (

x

y

) uses

x

-bit fixed-point arithmetic with

y

integer bits for CMA-ES.

Method	Accuracy (%)
Method	V2	R	Sketch	Avg.
ELaTTA	78.15	65.29	47.73	63.72
QELaTTA-V1	77.46	63.31	46.79	62.52
QELaTTA-V2 (8b5)	76.94	62.02	46.33	61.76
QELaTTA-V2 (8b4)	77.21	62.06	46.37	61.88
QELaTTA-V2 (8b3)	76.86	61.45	45.99	61.43

Table 19: Accuracy (%) of ELaTTA and its quantized variants on ImageNet-V2, ImageNet-R, and ImageNet-Sketch with ViT-Base under strict single-instance adaptation. QELaTTA-V2 (

x

y

) uses

x

-bit fixed-point arithmetic with

y

integer bits for CMA-ES.

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 322 papers

EmbodiTTA: Resource-Efficient Test-Time Adaptation for Embodied Visual Systems
May 2, 2025Xiao Ma, Young D. Kwon, Dong MaTest-Time Adaptation Continual Test-Time Adaptation
Singapore Management University, Singapore · Samsung AI Center–Cambridge, United Kingdom · University of Cambridge, United Kingdom
Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts
Oct 5, 2026Seungmin Oh, Seunghun Kang, Jongbin RyuTest-Time Adaptation Test-Time Adaptation of VLMs
Ajou University
StableTTA: Improving Vision Model Performance by Training-free Test-Time Adaptation Methods
Apr 6, 2026Zheng Li, Jerry Cheng, Huanying Helen GuTest-Time Adaptation Ensemble Learning
Department of Computer Science New York Institute of Technology

May 2, 2025J/K move · Enter open · S save

EmbodiTTA: Resource-Efficient Test-Time Adaptation for Embodied Visual Systems

Xiao Ma, Young D. Kwon, Dong Ma

Singapore Management University, Singapore · Samsung AI Center–Cambridge, United Kingdom · University of Cambridge, United Kingdom

Test-Time Adaptation Continual Test-Time Adaptation On-Device Training Distribution Shift Detection

Open paper AskPDF arXiv

Continual Test-time adaptation (CTTA) continuously adapts the deployed model on every incoming batch of data. While achieving optimal accuracy, existing CTTA approaches present poor real-world applicability on resource-constrained edge devices, due to the substantial memory overhead and energy consumption. In this work, we first introduce a novel paradigm -- on-demand TTA -- which triggers adaptation only when a significant domain shift is detected. Then, we present OD-TTA, an on-demand TTA framework for accurate and efficient adaptation on edge devices. OD-TTA comprises three innovative techniques: 1) a lightweight domain shift detection mechanism to activate TTA only when it is needed, drastically reducing the overall computation overhead, 2) a source domain selection module that chooses an appropriate source model for adaptation, ensuring high and robust accuracy, 3) a decoupled Batch Normalization (BN) update scheme to enable memory-efficient adaptation with small batch sizes. Extensive experiments show that OD-TTA achieves comparable and even better performance while reducing the energy and computation overhead remarkably, making TTA a practical reality.

Abstract

Test-Time Adaptation

Explore similar work

EmbodiTTA: Resource-Efficient Test-Time Adaptation for Embodied Visual Systems

Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts

StableTTA: Improving Vision Model Performance by Training-free Test-Time Adaptation Methods