IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation

Sunghyun Baek; Jaemyung Yu; Seunghee Koh; Minsu Kim; Hyeonseong Jeon; Junmo Kim

cs.CVMar 9, 2026

IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation

Authors: Sunghyun Baek, Jaemyung Yu, Seunghee Koh, Minsu Kim, Hyeonseong Jeon, Junmo Kim

Organizations: Korea Advanced Institute of Science and Technology (KAIST) · LG Energy Solution

Abstract

Test-time adaptation (TTA) has been widely explored to prevent performance degradation when test data differ from the training distribution. However, fully leveraging the rich representations of large pretrained models with minimal parameter updates remains underexplored. In this paper, we propose Intrinsic Mixture of Spectral Experts (IMSE) that leverages the spectral experts inherently embedded in Vision Transformers. We decompose each linear layer via singular value decomposition (SVD) and adapt only the singular values, while keeping the singular vectors fixed. We further identify a key limitation of entropy minimization in TTA: it often induces feature collapse, causing the model to rely on domain-specific features rather than class-discriminative features. To address this, we propose a diversity maximization loss based on expert-input alignment, which encourages diverse utilization of spectral experts during adaptation. In the continual test-time adaptation (CTTA) scenario, beyond preserving pretrained knowledge, it is crucial to retain and reuse knowledge from previously observed domains. We introduce Domain-Aware Spectral Code Retrieval, which estimates input distributions to detect domain shifts, and retrieves adapted singular values for rapid adaptation. Consequently, our method achieves state-of-the-art performance on various distribution-shift benchmarks under the TTA setting. In CTTA and Gradual CTTA, it further improves accuracy by 3.4 percentage points (pp) and 2.4 pp, respectively, while requiring 385 times fewer trainable parameters. Our code is available at https://github.com/baek85/IMSE.

Figures & tables

Figure 1: IMSE-Retrieval with domain bank. When adapting to a new domain, we select initial singular values based on domain similarity using the domain descriptor, then fine-tune the $\bm{\sigma}$ components within linear layers. The adapted spectral code $\bm{S}$ is stored in the Domain Bank, and this process repeats for subsequent domains. Note that domain descriptors are designed to estimate the distribution of test data.

Pretrain	Method	Noise			Blur				Weather				Digital				Avg.( ${\uparrow}$ )
Pretrain	Method	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elastic	Pixel.	JPEG	Avg.( ${\uparrow}$ )
Supervised	Source	46.9	47.6	46.9	42.7	34.2	50.5	44.7	56.9	52.6	56.5	76.1	31.8	46.7	65.5	66.0	51.0
	T3A	16.6	11.8	16.4	29.9	24.3	34.5	28.5	15.9	27.0	49.1	56.1	44.8	33.3	45.1	49.4	32.2
	CoTTA	40.3	31.8	39.6	35.5	33.1	46.9	37.3	2.9	46.4	59.1	71.7	55.5	46.4	59.4	59.0	44.4
	DDA	52.5	54.0	52.1	33.8	40.6	33.3	30.2	29.7	35.0	5.0	48.6	2.7	50.0	60.0	58.8	39.1
	MEMO	58.1	59.1	58.5	51.6	41.2	57.1	52.4	64.1	59.0	62.7	80.3	44.6	52.8	72.2	72.1	59.1

Table 1: Test-time adaptation on ImageNet-C (50k) . Accuracy(%)(

{\uparrow}

) over 15 corruptions at severity level 5 using ViT-Base models with different pretraining strategies. Results with * are reproduced by us.

Table 3

	Noise			Blur				Weather				Digital
ImageNet-C (5k)	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elastic	Pixel.	JPEG	Avg.( ${\uparrow}$ )
Source	46.8	46.7	46.7	44.1	31.1	48.6	44.5	54.3	50.4	54.9	75.3	31.3	45.2	65.8	66.1	50.1
TENT *	48.0	50.2	51.2	45.5	34.0	51.4	46.9	57.3	54.6	56.9	76.3	39.5	46.4	66.9	66.7	52.8
CoTTA *	47.0	47.1	47.3	44.7	31.6	49.0	45.2	54.8	51.0	55.5	75.9	32.0	45.9	66.1	66.7	50.7
ViDA *	50.5	56.3	56.5	50.8	42.6	56.5	52.2	61.6	58.3	63.0	77.1	55.2	49.8	67.1	68.0	57.7
IMSE Retrieval	57.3	60.4	59.7	58.0	57.8	60.7	60.8	68.2	66.3	68.5	78.2	59.9	67.1	74.1	69.6	64.4

Table 4: Continual test-time adaptation on ImageNet-C (5k) . Accuracy(%)(

{\uparrow}

) over 15 corruptions at severity level 5 using a supervised ViT-Base model. Results with * are reproduced by us.

Panel of Figure 2 — Figure 2: (a) Comparison of top- $R$ % vs. bottom- $R$ % singular value selection. (b) Feature diversity across various training methods. (c) Adaptation performance across various training methods. (d) Impact of the threshold $\tau$ on domain shift detection and adaptation performance. CE and TTA losses denote $\mathcal{L}_{\text{ce}}$ and $\mathcal{L}_{\text{entmin}}$ , respectively.

Method	$\alpha{=}1.0$ (Mild)	$\alpha{=}0.5$ (Moderate)	$\alpha{=}0.1$ (Severe)	$\alpha{=}0.05$ (More severe)
IMSE w/o $\mathcal{L}_{\text{dm}}$	67.79	67.78	67.64	67.33
IMSE	68.92	68.88	68.62	68.41
Improvement	+1.13	+1.10	+0.98	+1.08

Table 5: Test-time adaptation on ImageNet-C under non-i.i.d. setting. Average accuracy (%) across 15 corruption domains for different concentration factors

\alpha

Table 7

Table 8

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MAE	CLIP	ViT-Large
TENT	1e-3	1e-3	3e-4
SAR	1e-3	3e-3	1e-3
DPAL	4e-3	3e-3	4e-3
IMSE	1e-3	3e-3	4e-3

Appendix

Table 11: Learning rates for each method across different pretrained models.

	Noise			Blur				Weather				Digital
ImageNet-C (50k)	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elastic	Pixel.	JPEG	Avg.( ${\uparrow}$ )
Source	62.5	62.0	63.3	52.9	45.3	60.7	55.2	66.0	62.4	62.6	79.9	40.1	56.2	74.3	72.8	61.1
TENT *	65.2	66.0	66.1	60.8	54.4	64.1	60.4	68.2	63.7	65.9	80.4	58.0	60.3	75.9	73.6	65.5
SAR *	67.4	68.3	69.3	55.8	59.6	66.1	64.2	68.8	68.1	70.5	81.6	59.0	69.7	78.0	75.8	68.1
DPAL *	66.6	68.4	68.1	62.5	63.8	67.9	65.7	72.0	64.4	69.5	81.5	56.5	71.3	73.6	75.7	68.5
IMSE	70.1	71.0	71.0	68.1	68.5	71.8	71.7	75.8	74.0	76.3	82.7	68.5	76.0	80.4	78.3	73.6

Appendix

Table 12: Test-time adaptation on ImageNet-C (50k) . Accuracy (

{\uparrow}

) over 15 corruptions at severity level 5 using supervised pretrained ViT-Large model. Results with * are reproduced by us.

Attn.q	Attn.k	Attn.v	Attn.proj	MLP.fc1	MLP.fc2	Acc ( ${\uparrow}$ )
-	-	-	-	-	-	51.0
✓		-	-	-	-	59.6
-	✓	-	-	-	-	61.8
-	-	✓	-	-	-	66.6
✓	✓	✓	-	-	-	67.4
✓	✓	✓	✓	-	-	68.4

Appendix

Table 13: Ablation study on target modules. The best is bolded and the second best is underlined .

Panel of Figure 4 — Figure 4: (a) Diversity of alignment patterns in 3rd Transformer Block. (b) Diversity of alignment patterns in 6th Transformer Block. (c) Diversity of alignment patterns in 9th Transformer Block.

Figure 5: Domain distance matrix. Pairwise distance matrix among 15 domain descriptors.

Method	bit error	color quantization	far focus	flash	fog	h265 abr	h265 crf	iso noise	low light	near focus	xy motion blur	z motion blur	Avg.( ${\uparrow}$ )
Source *	20.3	53.1	61.7	47.3	39.4	52.8	57.3	59.1	56.3	70.5	43.2	47.4	50.7
TENT *	1.2	64.1	68.2	52.1	7.5	56.1	60.7	65.6	70.3	76.2	56.4	61.2	53.3
SAR *	13.8	64.7	68.7	54.3	41.4	57.1	61.4	66.3	70.8	76.4	57.8	62.5	57.9
DPAL *	13.1	64.2	68.3	53.7	43.4	57.0	61.2	65.8	70.0	75.8	56.6	61.7	57.6
IMSE	15.5	65.0	70.1	55.2	44.5	58.1	62.2	67.3	71.6	77.1	58.9	63.9	59.1

Appendix

Table 14: Test-time adaptation on ImageNet-3DCC (50k) . Accuracy(%)(

{\uparrow}

) over 11 corruptions at severity level 5 using supervised pretraining ViT-Base. Results with * are reproduced by us.

Method	R $\rightarrow$ A	R $\rightarrow$ C	R $\rightarrow$ P	A $\rightarrow$ R	A $\rightarrow$ C	A $\rightarrow$ P	C $\rightarrow$ R	C $\rightarrow$ A	C $\rightarrow$ P	P $\rightarrow$ R	P $\rightarrow$ A	P $\rightarrow$ C	Avg.( ${\uparrow}$ )
Source *	79.68	56.05	88.35	87.46	58.55	82.90	84.66	78.37	84.29	87.74	75.32	55.69	76.58
TENT *	79.68	56.08	88.91	88.01	59.61	83.10	85.42	79.68	85.01	87.90	75.48	55.71	77.05
SAR *	79.72	56.72	88.37	87.62	61.42	83.03	85.40	79.48	84.61	87.81	76.02	56.26	77.20
DPAL *	79.93	55.57	86.52	87.14	59.17	82.61	85.33	79.02	84.09	86.68	75.85	55.25	76.43
IMSE	81.04	56.35	88.91	88.13	60.36	83.44	85.74	79.39	84.88	88.04	76.14	56.28	77.39

Appendix

Table 15: Test-time adaptation on OfficeHome . Accuracy(%)(

{\uparrow}

) over 12 source–target domain pairs and average performance. Results with * are reproduced by us.

Method	R $\rightarrow$ C	R $\rightarrow$ P	R $\rightarrow$ S	C $\rightarrow$ R	C $\rightarrow$ P	C $\rightarrow$ S	P $\rightarrow$ R	P $\rightarrow$ C	P $\rightarrow$ S	S $\rightarrow$ R	S $\rightarrow$ C	S $\rightarrow$ P	Avg.( ${\uparrow}$ )
Source *	66.84	76.14	59.44	80.46	69.85	65.93	86.72	69.04	59.95	82.19	72.95	73.62	71.93
TENT *	68.25	76.45	60.36	81.91	72.32	67.01	87.60	69.97	59.21	82.76	73.05	73.11	72.67
SAR *	69.64	77.78	64.23	81.91	72.83	67.73	86.72	70.77	59.41	82.77	73.53	73.41	73.39
DPAL *	68.14	77.54	60.86	81.78	72.79	66.07	84.93	70.16	58.86	81.61	72.54	70.11	72.12
IMSE	69.33	77.78	62.40	82.45	73.23	67.57	87.85	70.42	59.72	83.10	73.73	73.55	73.43

Appendix

Table 16: Test-time adaptation on DomainNet . Accuracy(%)(

{\uparrow}

) over 12 source–target domain pairs and average performance. Results with * are reproduced by us.

Method	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	Avg.( ${\uparrow}$ )
Source *	49.2	50.2	50.4	50.0	49.8	50.4	50.7	49.8	50.4	50.0	50.1
TENT *	53.3	56.9	57.8	57.8	57.5	58.7	59.2	57.9	57.9	58.1	57.5
CoTTA *	49.7	52.1	53.1	53.1	53.2	54.0	54.6	53.5	54.0	54.0	53.1
ViDA *	53.7	57.2	59.3	58.7	59.1	59.5	59.4	59.7	59.6	59.8	58.6
IMSE-Retrieval	63.7	65.1	65.8	65.2	64.8	65.9	65.0	65.4	65.1	65.3	65.1

Appendix

Table 17: Continual test-time adaptation on ImageNet-C under the domain-recurring setting. Average accuracy (%) for each round (R1–R10) and the average over all rounds. Results with * are reproduced by us.

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 322 papers

SPeaR: Test-Time Adaptation with Steering Primitives for Realigning Representations
Sep 21, 2026Muhammad Sudipto Siam Dip, Ali EtemadTest-Time Adaptation Continual Test-Time Adaptation
Department of Electrical and Computer Engineering Queen’s University, Canada
DOME: Learning Transferable Domain Variables from Sparse Supervision for Test-Time Adaptation
Jun 2, 2026Xiaoran Xu, Yifan Xu, Yupeng Wu +2Representation Learning Test-Time Adaptation
MAIS, IACAS · Beijing, China
Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts
Oct 5, 2026Seungmin Oh, Seunghun Kang, Jongbin RyuTest-Time Adaptation Test-Time Adaptation of VLMs
Ajou University

Sep 21, 2026J/K move · Enter open · S save

SPeaR: Test-Time Adaptation with Steering Primitives for Realigning Representations

Muhammad Sudipto Siam Dip, Ali Etemad

Department of Electrical and Computer Engineering Queen’s University, Canada

Test-Time Adaptation Continual Test-Time Adaptation

Open paper AskPDF arXiv

Test-time adaptation (TTA) addresses distribution shift using only unlabeled test data. Existing methods typically adapt pretrained models by updating their parameters, limiting both what is adapted and where adaptation can occur within the network. We instead keep the pretrained network frozen and steer its intermediate representations. We introduce SPeaR (Steering Primitive for Realigning Representations), which inserts lightweight learnable modules at stage boundaries and optimizes them directly from the test stream, requiring neither source data nor supervised warm-up. Each primitive is optimized using a gated objective that reduces uncertainty only when adaptation is beneficial, along with a diversity regularizer to prevent collapse, and a multi-depth anchor to stabilize adaptation. We show that steering early representations is the most effective strategy, and that the same primitive transfers across convolutional and Transformer architectures. Across CIFAR-10-C, CIFAR-100-C, and ImageNet-C, SPeaR consistently matches or outperforms methods that adapt orders of magnitude more parameters, remains robust across a wide range of batch sizes, and preserves source-domain performance during continual adaptation.

Abstract

Test-Time Adaptation

Explore similar work

SPeaR: Test-Time Adaptation with Steering Primitives for Realigning Representations

DOME: Learning Transferable Domain Variables from Sparse Supervision for Test-Time Adaptation

Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts