The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation

Guannan Lai; Da-Wei Zhou; Zhenguo Li; Han-Jia Ye

cs.CVMar 23, 2026

The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation

Authors: Guannan Lai, Da-Wei Zhou, Zhenguo Li, Han-Jia Ye

Organizations: School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong University of Science and Technology · Frontier Robotics

Abstract

Continual Test-Time Adaptation (CTTA) aims to enable models to adapt online to unlabeled data streams under distribution shift without accessing source data. Existing CTTA methods face an efficiency-generalization trade-off: updating more parameters improves adaptation but severely reduces online inference efficiency. An ideal solution is to achieve comparable adaptation with minimal feature updates; we call this minimal subspace the golden subspace. We prove its existence in a single-step adaptation setting and show that it coincides with the row space of the pretrained classifier. To enable online maintenance of this subspace, we introduce the sample-wise Average Gradient Outer Product (AGOP) as an efficient proxy for estimating the classifier weights without retraining. Building on these insights, we propose Guided Online Low-rank Directional adaptation (GOLD), which uses a lightweight adapter to project features onto the golden subspace and learns a compact scaling vector while the subspace is dynamically updated via AGOP. Extensive experiments on classification and segmentation benchmarks, including autonomous-driving scenarios, demonstrate that GOLD attains superior efficiency, stability, and overall performance. Our code is available at https://github.com/AIGNLAI/GOLD.

Figures & tables

Panel of Figure 1 — Figure 1: Results on the CIFAR100-C dataset. Left: performance–efficiency distribution of evaluated methods (TCA [ 33 ] is closed-source and shown as a line). Right: accuracy over the continuous test-time adaptation process; shaded background bands indicate periods of domain shift. In the golden-shaded intervals, prior methods exhibit significant performance degradation, whereas our method maintains generalization.

Panel of Figure 2 — Figure 2: (a) Alignment between AGOP-derived subspace and source-derived ground-truth subspace as testing progresses. AGOP rapidly converges and remains highly aligned. (b) AGOP spectrum cumulative energy: the top 64–128 eigenvectors capture over 99% of the energy, indicating strong low-rank concentration.

Figure 3: Overview of GOLD: it maintains an online AGOP estimator, from which the top- $r$ eigenvectors $V_{t}$ are extracted via eigendecomposition to form the golden subspace. A residual low-rank adapter of rank $r$ is then applied to learn a lightweight scaling vector $S_{t}$ . Meanwhile, a combination of self-training loss and prototype-based contrastive loss provides stable supervision.

Figure 4: Comparison of method efficiency. The vertical axis shows the time required for each method to perform adaptation and prediction in each test batch.

$W^{T}W$ init	AGOP	$\mathcal{L}_{\mathrm{cont}}$	CIFAR10-C	CIFAR100-C	ImageNet-C
-	-	-	16.24	29.87	62.45
$\surd$	-	-	15.11	29.15	60.24
$\surd$	-	$\surd$	14.64	28.63	60.01
$\surd$	$\surd$	-	15.06	28.77	59.87
-	$\surd$	$\surd$	14.32	28.61	59.52
$\surd$	$\surd$	$\surd$	14.12	28.56	59.32

Table 2: Ablation studies of two components (

G_{t}

and

S_{t}

) on the CIFAR10-C, CIFAR100-C, and ImageNet-C datasets.

Figure 5: A visual comparison of segmentation results from different adaptive methods on the CarlaTTA sequence is presented. White boxes indicate areas where GOLD has made progress, and magnified images are shown in the supplementary material.

	day2night	clear2fog	clear2rain	dynamic	highway
Source	58.4	52.8	71.8	46.2	24.7
MEMO [ 53 ]	61.0	55.1	71.6	36.7	23.6
TENT [ 42 ]	61.5	56.0	70.9	50.8	32.8
CoTTA [ 44 ]	61.4	56.8	70.7	46.3	33.8
GOLD	61.8	57.1	71.0	48.9	34.5

Table 3: Quantitative results of semantic segmentation under gradual domain shifts on CarlaTTA. We report mIoU (%) on five test sequences. Best results are in bold , second best are underlined .

Symbol	Meaning
$D_{S}$	Labeled source-domain dataset
$D_{T}$	Unlabeled target-domain data stream
$X_{t}$	Mini-batch of target samples at step $t$
$g_{\theta}$	Full model composed of feature extractor and classifier
$g_{\phi}$	Pretrained feature extractor (frozen backbone)
$h_{\psi}$	Classifier head producing logits

Table S1: Notation Summary.

Figure S1 : Hyper-parameter robustness: representative results from the sweep described in Section C.3 . Each subplot shows the evaluated metric as a function of a single hyper-parameter while other hyper-parameters are held near their default values.

Figure S2 : Detailed segmentation comparisons (cropped views). White boxes indicate regions shown at higher resolution to emphasize the qualitative gains of GOLD in challenging conditions (shadows, reflections, fog/night).

	CIFAR10-C				CIFAR100-C				ImageNet-C
Method	4	16	32	64	4	16	32	64	4	16	32	64
TENT [ 42 ]	86.9	61.9	40.2	22.9	98.3	96.2	92.9	86.1	99.0	82.5	64.5	62.6
Ada [ 9 ]	34.5	19.7	17.7	17.4	90.1	54.6	41.3	35.4	98.4	81.9	69.9	65.5
CoTTA [ 44 ]	52.6	26.9	21.3	17.5	68.8	39.1	36.7	34.3	97.5	82.0	64.8	62.7
RMT [ 13 ]	51.4	26.5	20.0	16.7	87.4	48.7	39.1	32.7	99.4	83.4	68.7	60.2
SANTA [ 8 ]	39.5	19.5	17.5	16.8	58.5	35.8	33.3	32.5	98.7	68.0	62.8	60.1

Table S2 : Performance under different test-time batch sizes. Results are shown for CIFAR-10-C, CIFAR-100-C and ImageNet-C. Columns report the evaluated batch size. The best value in each column (among the listed methods) is typeset in bold .

Method	Trainable parameter ratio (%)	FLOPs (G)	Peak GPU memory (GB)
CoTTA	100	7842.80	10.88
RMT	100	1886.02	5.07
GTTA	100	2614.27	5.21
SANTA	0.365	1348.09	4.61
GOLD	0.373	1425.14	5.37

Table S3 : Comprehensive efficiency comparison.

Method	CoTTA	RMT	BeCoTTA	SANTA
Error (%)	16.21	16.73	16.28	16.29
Method	EATA	SAR	ViDA	GOLD
Error (%)	17.84	20.31	17.24	14.15

Table S4 : Long-term generalization comparison on CIFAR10-C under 10 rounds of continual adaptation. Lower is better.

Continual Test-Time Adaptation

Continual test-time adaptation updates pretrained models on unlabeled, changing deployment data while retaining useful knowledge across shifts.

See all 72 papers

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions
Jul 9, 2026Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang +6Source-Free Domain Adaptation Distribution Shift Robustness
The University of Texas at Dallas, USA · LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS), McGill - ÉTS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec · Tulane University, USA +2
TestDG: Test-time Domain Generalization for Continual Test-time Adaptation
Apr 7, 2025Sohyun Lee, Nayeong Kim, Juwon Kang +2Continual Test-Time Adaptation Cross-Domain Generalization
POSTECH · GenGenAI · KAIST AI
Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptation
Sep 29, 2026Youjia Zhang, Huiling Liu, Soyun Choi +2Adaptive Inference Test-Time Adaptation
Sungkyunkwan University · Nanyang Technological University

Jul 9, 2026J/K move · Enter open · S save

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang, Shuaicheng Niu+5

The University of Texas at Dallas, USA · LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS), McGill - ÉTS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec · Tulane University, USA+2

Source-Free Domain Adaptation Distribution Shift Robustness Test-Time Adaptation Continual Test-Time Adaptation

Open paper AskPDF arXiv

Deep neural nets achieve remarkable performance when training and test data share the same distribution, but this assumption frequently breaks in real-world deployment, where data undergoes continual distributional shifts. Continual Test-Time Adaptation (CTTA) addresses this challenge by adapting pretrained models to non-stationary target distributions on-the-fly, without access to source data or labeled targets, while mitigating two critical failure modes: catastrophic forgetting of source knowledge and error accumulation from noisy pseudo-labels over extended time horizons. In this comprehensive survey, we formally define the CTTA problem, analyze the diverse continual domain shift patterns that characterize different evaluation protocols, and propose a hierarchical taxonomy that categorizes existing methods into three families: optimization-based strategies (entropy minimization, pseudo-labeling, parameter restoration), parameter-efficient methods (normalization layer adaptation, adaptive parameter selection), and architecture-based approaches (teacher-student frameworks, adapters, visual prompting, masked modeling). We systematically review representative methods within each category and present comparative benchmarks and experimental results across standard evaluation settings. Finally, we discuss the limitations of current approaches and highlight emerging research directions, including the adaptation of foundation models and black-box systems, thereby providing a roadmap for future research in robust continual test-time adaptation.

Abstract

Continual Test-Time Adaptation

Explore similar work

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

TestDG: Test-time Domain Generalization for Continual Test-time Adaptation

Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptation