Sparse MLLM Anchors, Dense Adaptation: Breaking the Self-Referential Loop in Wild Test-Time Adaptation

Real-world deployment often exposes vision models to distribution shifts, making test-time adaptation (TTA) important for reliable prediction. Existing TTA methods can be difficult to deploy on edge devices because they require backpropagation, activation storage, or test-time batches, which increase computation, memory, and state. We propose ELaTTA (Efficient Latent Test-Time Adaptation), a forward-only, gradient-free framework for single-instance TTA. ELaTTA freezes the pretrained model and adapts each test input by searching a low-dimensional coefficient vector in a principal latent subspace computed offline from a source-domain cache. The basis is stored once and source data are discarded before deployment. At test time, CMA-ES performs covariance-adaptive Gaussian search, ranks candidate corrections by prediction entropy, and selects the best candidate within a fixed evaluation budget. On five visual benchmarks spanning corruption, natural, rendition, sketch, and cross-domain shifts, ELaTTA provides strong accuracy under strict and continual single-instance protocols. In the strict single-instance comparison, MEMO requires \emph{64.7$\times$} the computation and \emph{12.4$\times$} the peak memory of ELaTTA. A complementary keyword-spotting experiment shows that the same coefficient-search design transfers beyond vision. We further demonstrate ViT-Base adaptation on a Raspberry Pi 5, confirming the feasibility of deploying ELaTTA on an edge device.

Vision-language models transfer well in zero-shot settings, but at deployment the visual and textual branches often shift asymmetrically. Under this condition, entropy-based test-time adaptation can sharpen the fused posterior while increasing error, because an unreliable modality may still dominate fusion. We study this failure mode through a majorization view of multimodal posteriors and cast adaptation as a constrained de-mixing problem on the fused prediction. Based on this view, we propose MG-MTTA, which keeps the backbone frozen and updates only a lightweight gate or adapter. The objective combines fused-posterior entropy minimization with a reliability-aware gate prior built from anchor-based modality consistency and cross-modal conflict. Our analysis gives conditions under which entropy reduction preserves the correct ranking and a threshold that characterizes modality-dominance failure. On the ImageNet-based benchmark, MG-MTTA improves top-1 accuracy from 57.97 to 66.51 under semantics-preserving textual shift and from 21.68 to 26.27 under joint visual-textual shift, while remaining competitive in the visual-only benchmark. These results show that multimodal test-time adaptation should control modality reliability, not just prediction entropy.

Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements. We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image-text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation. Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU. Project page: https://lingrayy.github.io/MSSA/ .