cs.CVNov 10, 2026

What Entropy-Based View Selection Cannot See: An Audit of Test-Time Adaptation for Vision-Language Models

Authors: Sivasuthan Sukumar, Uthayasanker Thayasivam

Organizations: University of Moratuwa Sri Lanka

Abstract

Vision-language models such as CLIP can classify images into categories they were never explicitly trained to recognize, but accuracy degrades when test images differ in style or source from their training data. A common remedy generates randomly cropped and flipped copies of each test image, called views, and combines their predictions. Since some views miss the object, methods must identify which views to trust. The dominant approach uses prediction entropy to measure confidence and retains the most certain views. Other methods use signals such as embedding typicality. Despite repeated refinements of these scoring functions, the information they rely on, the model's own response to each view, remains largely unexamined. We investigate this choice while holding other pipeline stages fixed. First, we show that view selection, rather than the gradient update, accounts for almost all accuracy gains. We then examine whether entropy identifies useful views when predictions are wrong, despite at least one view predicting the correct class. In these cases, entropy ranks correct views above incorrect ones no better than chance, recovering only about one-twentieth of the accuracy recovered by an oracle over the same views. This finding holds across two image encoders and under a control ruling out an obvious statistical artifact. An exact analysis explains this limitation: entropy measures confidence but not which class a view favors. We identify an unused signal already available in the augmentation pipeline: the fraction of the original image retained by each crop. Ranking views by this quantity improves accuracy on 14 of 15 benchmark datasets over entropy and on 11 of 15 over the strongest scoring function tested. A pre-method measurement also predicts which datasets benefit.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models

    Aug 6, 2026Jingyan Jiang, Yaru Sun, Xiao Chen +5VLM AdaptationTest-Time Adaptation of VLMs

  2. To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models

    Sep 8, 2026Siru Jiang, Yuwei Liang, Jian Liang +2Test-Time AdaptationVLM Adaptation

  3. ENCORE: Entropy-Guided Cropping and Attention Regularization for Robust Vision--Language Understanding

    Aug 24, 2026Yuanhao Sun, Huawei Ji, Jiaxin Ding +2VLM AdaptationVLM Robustness