Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference. Although significant efforts are devoted to adapting VLMs at test time, they rely heavily on noisy pseudo-labels predicted directly from raw embedding similarities during inference, which are unreliable under distribution shift and mislead the adaptation. To avoid noise amplification, existing works craft coarse-grained surrogate objectives during adaptation, which fail to explicitly model sample-level relationships across different modalities, creating objective mismatch with inference, thus leading to marginal performance improvement. In this work, we aim to bridge the detached objectives of inference and adaptation for VLMs, and propose a principled VLM TTA method called \algname. For VLM inference, we formulate the zero-shot image classification task as a cross-modal alignment problem encoded via a Wasserstein OT formulation, providing robust pseudo-labels at the sample-level to effectively adapt VLMs. For VLM adaptation, we adopt a soft-label InfoNCE loss to adapt VLMs based on the OT-induced pseudo-labels, leveraging fine-grained supervisions to explicitly model relationships of individual image-text pairs via contrastive learning, which empowers accurate inference at the same granularity. Moreover, we theoretically reveal that the InfoNCE loss can be neatly reformulated as a Wasserstein OT formulation, thereby unifying the objectives of the inference and adaptation of VLMs to achieve their mutual benefits. Extensive experiments demonstrate the effectiveness and efficiency of our methods, outperforming the best-performing methods by up to 7% with state-of-the-art efficiency.
Figures & tables
Figure 1: Training dynamics of different combinations of inference and adaptation objective on CIFAR10-C. (1) Embedding-based inference are consistently outperformed by OT-based inference under the same loss function, and may even leads to performance degradation during adaptation due to excessive noise amplification; (2) ORIGIN adopts InfoNCE loss with OT-induced pseudo-labels to explicitly model sample-level relationships across modalities, increasing accuracy monotonically during adaptation and consistently achieves the best performance.
Figure 2: An overview of ORIGIN . VLMs first encode test images via the image encoder and text descriptions of different classes via the text encoder. During inference , ORIGIN infers zero-shot image classification by solving an entropic Wasserstein OT problem. During adaptation , ORIGIN explicitly models the embedding relationships of shifted modalities at the sample-level via a soft-label InfoNCE loss, and optimizes normalization layers of the image encoder.
Dataset
Method
Noise
Blur
Weather
Digital
Mean
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elast.
Pixel.
JPEG
CIFAR10-C
Source
37.7
42.0
54.2
71.8
40.9
68.1
74.0
74.1
77.1
70.6
84.0
62.7
53.6
47.2
59.0
61.2
TDA
40.9
44.0
50.5
72.3
44.6
71.7
75.6
75.5
78.2
71.6
85.7
62.4
56.3
48.3
58.1
61.9
DMN
43.0
47.4
53.8
73.5
40.0
71.9
76.3
74.5
76.2
71.0
84.2
60.2
54.4
47.2
57.0
62.4
VTE
42.8
46.7
63.9
71.0
45.6
69.0
73.6
76.7
78.5
71.0
85.7
57.5
59.5
60.8
61.7
64.3
ZERO
38.7
43.9
57.3
71.7
40.9
69.0
74.5
74.5
77.6
72.7
84.1
60.3
55.5
48.6
61.8
61.6
Table 1: Benchmarking results with ViT-B-16. The 1st / 2nd best results are highlighted in bold and underline , respectively. Additional benchmarking results with ViT-B-32 are shown in Table 4
Figure 3: Embedding visualization of different VLM TTA methods by t-SNE on CIFAR10-C. Stars denote text embeddings of different classes, and round points denote image embeddings.
Figure 5: Hyperparameter sensitivity study of ORIGIN on CIFAR10-C
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameters
α
τ / ϵ
CIFAR10-C
0.5
0.01
CIFAR100-C
0.5
0.01
ImageNet-C
0.3
0.01
Appendix
Table 3: Detailed hyperparameter settings of ORIGIN .
Dataset
Method
Noise
Blur
Weather
Digital
Mean
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elast.
Pixel.
JPEG
CIFAR10-C
Source
36.0
40.0
43.5
70.1
41.4
64.3
69.8
70.8
72.6
66.6
81.3
64.6
59.3
48.6
56.9
59.3
TDA
41.8
43.2
41.7
71.3
44.7
66.9
72.4
72.6
74.7
68.4
83.5
66.1
62.9
50.9
55.8
61.1
DMN
38.9
39.3
42.5
70.6
39.2
65.1
72.7
72.3
73.4
66.4
82.8
58.6
60.7
49.1
56.3
59.1
VTE
47.7
49.8
53.6
71.6
54.2
67.9
72.9
76.9
76.5
70.9
83.7
60.9
69.2
58.7
61.2
65.3
ZERO
36.9
41.6
46.8
70.8
42.1
65.3
71.7
71.9
73.4
68.8
83.0
65.3
62.4
47.7
59.0
60.4
Appendix
Table 4: Benchmarking results with ViT-B-32. The 1st / 2nd best results are highlighted in bold and underline , respectively. The proposed ORIGIN consistently achieves the best performance across all benchmarks, with up to 10% improvement in mean accuracy.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
Test-time adaptation (TTA) has emerged as a prominent strategy for adapting vision-language models to distribution shifts during inference. We conduct a per-sample analysis of model predictions before and after adaptation, and observe two failure modes in existing TTA methods that echo previous work. Adaptations are frequently negligible, yielding no change in the model's predictions, and more severely, they can be detrimental by flipping previously correct predictions to incorrect ones. This naturally raises a question: Can we identify and skip such negligible or harmful adaptations? In this work, we introduce a new problem of selective adaptation, which aims to determine whether a given test sample should undergo adaptation or be skipped. To this end, we propose Cross-Augmentation Similarity (CAS), a simple baseline that performs adaptation only when predictions across augmented views exhibit low similarity. Notably, CAS not only preserves but in some cases improves overall accuracy, even when skipping nearly 85% of the adaptation process. We hope other researchers will explore this new direction and surpass the performance of our baseline. Our code is available at https://github.com/sirujiang/selective-adaptation.
Siru Jiang, Yuwei Liang, Jian Liang +2
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China +1
Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.
Yan Huang, Guowei Wang, Xu Wang +2
Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China +1
Vision-language models (VLMs) have achieved impressive performance on multimodal inference tasks, but the cost remains a significant challenge due to the large number of vision tokens processed during the prefill stage. Existing token pruning methods often rely on utilizing the static attention patterns directly, failing to exploit the dynamic internal signals within VLMs. To address the issue, we propose AdaptInfer, a novel plug-and-play framework for vision token pruning. First, we introduce a dynamic text-guided pruning mechanism that construct soft priors over text-token importance on each pruning layer, allowing more informed scoring of vision tokens at each stage. Second, we observe a highly consistent distribution of cross-modal attention shifts by architecture, which inspires us to introduce a efficient data-driven schedule which determines the pruning locations automatically. Experimental results have verified the effectiveness and generalization of the proposed method. Under the same token budget, AdaptInfer surpasses existing approaches in accuracy. For instance, AdaptInfer maintains averagely 99.4% accuracy on Qwen2-VL while 70% of the prefilling vision token overhead is reduced. The source code is available on: https://github.com/weiczh02/AdaptInfer-base.
Weichen Zhang, Zhui Zhu, Ningbo Li +5
Global Innovation Exchange, Tsinghua University, Beijing, China · Department of Automation, Tsinghua University, Beijing, China · School of Computer Science, Peking University, Beijing, China