Sparse MLLM Anchors, Dense Adaptation: Breaking the Self-Referential Loop in Wild Test-Time Adaptation

Zhenbin Wang; Lei Zhang; Lituan Wang; Yan Wang; Zhao Zhang; Wei Huang

cs.AISep 15, 2026

Sparse MLLM Anchors, Dense Adaptation: Breaking the Self-Referential Loop in Wild Test-Time Adaptation

Authors: Zhenbin Wang, Lei Zhang, Lituan Wang, Yan Wang, Zhao Zhang, Wei Huang

Organizations: Sichuan University

Abstract

Wild test-time adaptation (WTTA) updates a source model online under small test batches, concurrent distribution shifts, and time-varying class imbalance. Most WTTA methods derive their adaptation signals, including predictive uncertainty, sample reliability, and local feature geometry, from the model being adapted. When the source model is unreliable under shift, these signals can reinforce its own errors, forming a self-referential loop. We introduce MASA (Multimodal-LLM-Anchored Semantic Adaptation), which complements model-internal evidence with structured semantic descriptions from a frozen multimodal large language model (MLLM). To limit inference cost, MASA queries the MLLM only for a small set of diverse, reliability-ranked anchors. The resulting descriptions capture the object family and nuisance factors such as style, viewpoint, and occlusion. MASA encodes these descriptions, propagates them to neighboring test samples, and stores the resulting visual-semantic information in an online prototype memory. Descriptor-aware retrieval from this memory provides an auxiliary target for lightweight adaptation of normalization-affine parameters. We evaluate MASA on the WTTA ImageNet-C benchmark under limited-batch, mixed-domain, and imbalanced-label-shift settings with ResNet and ViT backbones.

Figures & tables

Figure 1: Comparison of adaptation evidence. Left: many WTTA methods estimate reliability and construct adaptation targets from the model being adapted, allowing prediction errors to influence subsequent updates. Right: MASA queries a frozen MLLM on sparse anchors, propagates the encoded descriptors through local feature neighborhoods, and stores them in a visual–semantic prototype memory for descriptor-aware retrieval and memory admission.

Figure 2: MASA overview. Gray modules form the regional-evidence path. MASA ranks buffered observations, selects diverse anchors, and queries a frozen MLLM for object and nuisance descriptors. It propagates these descriptors to nearby samples and stores them in a bounded visual–semantic memory for descriptor-aware retrieval. Retrieved prototypes provide an auxiliary consistency target for normalization-affine adaptation.

Figure 3: From an image-only MLLM query to descriptor phrases. A fixed task-agnostic prompt returns structured fields. Each value is indexed by $l$ and denoted by $\xi_{l}$ ; list entries remain separate, while gray scores determine $\kappa_{a}$ and are not embedded.

Figure 4: Feature-space visualization after online adaptation. t-SNE features of 10 classes produced by ReCAP and MASA on ImageNet-C under Elastic Transform at severity level 5. Colors indicate classes, and boxes mark representative regions for comparison.

Table 5

ResNet50-GN				ViT-Base-LN
Method	Lv.5	Lv.4	Avg	Method	Lv.5	Lv.4	Avg
Source	30.6	42.7	36.7	Source	29.9	42.9	36.4
MEMO	31.2	43.0	37.1	MEMO	39.1	51.3	45.2
DDA	35.1	43.6	39.4	DDA	36.1	45.1	40.6
Tent	13.4	20.6	17.0	Tent	16.5	64.3	40.4
EATA	38.1	47.7	42.9	EATA	55.7	63.7	59.7

Table 3: Mixed-domain evaluation on ImageNet-C. We report accuracy (%) for ResNet50-GN and ViT-Base-LN at corruption severities 5 and 4. Avg averages the results across the two severities.

Config	RN50	ViT	Loss terms	RN50	ViT
Full	48.2	64.0	Visual only	46.8	63.3
$-$ proto-loss	46.5	63.2	$+$ predictive	47.4	63.6
$-$ propagation	47.2	63.6	$+$ semantic	48.2	64.0
$-$ reliability	46.9	63.5
$-$ PLPD	47.6	63.8

Table 4: Component ablation and prototype-discrepancy decomposition under label shift (average accuracy, %).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Hyperparameter sensitivity under label shift. Average accuracy (%) with ResNet50-GN when varying (a) the prototype-loss weight, (b) the periodic refresh interval, and (c) the memory capacity. Each sweep changes one parameter while fixing the others. Red squares mark the default settings, and the dashed line shows the ReCAP baseline under the same protocol.

Semantic source	ResNet50-GN			ViT-Base-LN
	bs1	mix	lbl	bs1	mix	lbl
None	47.0	46.7	47.4	65.9	63.3	63.6
Hash (text)	47.2	46.8	47.3	66.0	63.2	63.5
BLIP-2	47.7	47.0	47.7	66.2	63.4	63.7
LLaVA-1.5-7B	48.1	47.2	47.9	66.3	63.5	63.8
Qwen2-VL-2B	48.4	47.4	48.2	66.5	63.7	64.0

Appendix

Table 5: Effect of the semantic source on average accuracy (%) under each WTTA protocol. Here, bs1, mix, and lbl denote limited-batch, mixed-domain, and label-shift evaluation.

Hint to MLLM	RN50	ViT
None	48.2	64.0
Predicted class	47.9	63.8
Ground-truth class (oracle)	49.1	64.6

Appendix

Table 6: Effect of class information in the MLLM prompt. Results are average accuracy (%) under label shift. MASA uses the no-hint setting.

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 325 papers

Efficient Test-Time Adaptation through Latent Subspace Coefficients Search
Oct 13, 2025Xinyu Luo, Jie Liu, Kecheng Chen +6Test-Time Adaptation Continual Test-Time Adaptation
Department of Electrical Engineering, City University of Hong Kong, Hong Kong SAR, China
Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift
Apr 27, 2026Lixian Chen, Yanhui Chen, Junyi LinVLM Robustness Test-Time Adaptation
Guangdong University of Technology · Guangzhou, China
Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
Jul 20, 2026Lingrui Li, Nan Pu, Dong Zhao +4Memory-Augmented VLMs VLM Adaptation
School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore

Oct 13, 2025J/K move · Enter open · S save

Efficient Test-Time Adaptation through Latent Subspace Coefficients Search

Xinyu Luo, Jie Liu, Kecheng Chen, Yihua Shao+5

Department of Electrical Engineering, City University of Hong Kong, Hong Kong SAR, China

Test-Time Adaptation Continual Test-Time Adaptation

Open paper AskPDF arXiv

Real-world deployment often exposes vision models to distribution shifts, making test-time adaptation (TTA) important for reliable prediction. Existing TTA methods can be difficult to deploy on edge devices because they require backpropagation, activation storage, or test-time batches, which increase computation, memory, and state. We propose ELaTTA (Efficient Latent Test-Time Adaptation), a forward-only, gradient-free framework for single-instance TTA. ELaTTA freezes the pretrained model and adapts each test input by searching a low-dimensional coefficient vector in a principal latent subspace computed offline from a source-domain cache. The basis is stored once and source data are discarded before deployment. At test time, CMA-ES performs covariance-adaptive Gaussian search, ranks candidate corrections by prediction entropy, and selects the best candidate within a fixed evaluation budget. On five visual benchmarks spanning corruption, natural, rendition, sketch, and cross-domain shifts, ELaTTA provides strong accuracy under strict and continual single-instance protocols. In the strict single-instance comparison, MEMO requires \emph{64.7

\times

} the computation and \emph{12.4

\times

} the peak memory of ELaTTA. A complementary keyword-spotting experiment shows that the same coefficient-search design transfers beyond vision. We further demonstrate ViT-Base adaptation on a Raspberry Pi 5, confirming the feasibility of deploying ELaTTA on an edge device.

Abstract

Test-Time Adaptation

Explore similar work

Efficient Test-Time Adaptation through Latent Subspace Coefficients Search

Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation