What Entropy-Based View Selection Cannot See: An Audit of Test-Time Adaptation for Vision-Language Models
Organizations: University of Moratuwa Sri Lanka
Abstract
Vision-language models such as CLIP can classify images into categories they were never explicitly trained to recognize, but accuracy degrades when test images differ in style or source from their training data. A common remedy generates randomly cropped and flipped copies of each test image, called views, and combines their predictions. Since some views miss the object, methods must identify which views to trust. The dominant approach uses prediction entropy to measure confidence and retains the most certain views. Other methods use signals such as embedding typicality. Despite repeated refinements of these scoring functions, the information they rely on, the model's own response to each view, remains largely unexamined. We investigate this choice while holding other pipeline stages fixed. First, we show that view selection, rather than the gradient update, accounts for almost all accuracy gains. We then examine whether entropy identifies useful views when predictions are wrong, despite at least one view predicting the correct class. In these cases, entropy ranks correct views above incorrect ones no better than chance, recovering only about one-twentieth of the accuracy recovered by an oracle over the same views. This finding holds across two image encoders and under a control ruling out an obvious statistical artifact. An exact analysis explains this limitation: entropy measures confidence but not which class a view favors. We identify an unused signal already available in the augmentation pipeline: the fraction of the original image retained by each crop. Ranking views by this quantity improves accuracy on 14 of 15 benchmark datasets over entropy and on 11 of 15 over the strongest scoring function tested. A pre-method measurement also predicts which datasets benefit.
Figures & tables
| ImageNet and distribution shifts | Fine-grained | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Selection rule | IN | IN-A | IN-K | IN-V2 | IN-R | Avg. | Flwr | DTD | Pets | Cars | UCF | Cal | Food | SUN | Air | ESAT | All |
| Shannon entropy ( Shu et al., 2022 ) | 68.5 | 59.6 | 46.4 | 61.5 | 77.3 | 62.7 | 67.0 | 45.9 | 86.0 | 66.8 | 67.0 | 92.6 | 84.7 | 60.9 | 24.4 | 39.4 | 63.2 |
| Tsallis | 68.9 | 60.8 | 47.0 | 61.6 | 77.8 | 63.2 | 66.8 | 46.2 | 87.0 | 67.3 | 67.0 | 92.7 | 84.5 | 61.4 | 24.2 | 40.2 | 63.6 |
| Crop geometry (ours) | 68.9 | 59.9 | 47.8 ∗ | 62.0 | 78.7 ∗ | 63.4 | 68.0 ∗ | 45.1 | 88.8 ∗ | 67.8 | 67.6 | 94.2 ∗ | 85.5 ∗ | 61.2 | 25.1 | 42.5 ∗ | 64.2 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Score, and images it is evaluated on | ViT-B/16 | RN50 |
|---|---|---|
| Entropy, all images | ||
| Entropy, reducible errors (controlled) | ||
| Entropy, reducible errors (uncontrolled) | ||
| Image-feature statistics, reducible errors | ||
| Crop area, reducible errors | — |
| Dataset | Backbone | Achieved | Oracle | Headroom |
|---|---|---|---|---|
| ImageNet-A | ViT-B/16 | |||
| ImageNet-A | RN50 | |||
| ImageNet | ViT-B/16 | |||
| ImageNet | RN50 | |||
| ImageNet-R | ViT-B/16 | |||
| DTD | ViT-B/16 |
| Identity | Residual |
|---|---|
| Logit-space gradient of | |
| Equation 3 , first form | |
| Equation 3 , pairwise form | |
| Hessian of | |
| Tangent eigenvalues at the uniform point ( ) | |
| Suppression peak at | 5 s.f. |
| Backbone | Dataset | Entropy | Sorted | Unsorted | Oracle | |||
|---|---|---|---|---|---|---|---|---|
| MetaCLIP-B16 | ImageNet-A | 200 | 496 | |||||
| MetaCLIP-B16 | ImageNet | 1000 | 359 | |||||
| MetaCLIP-B16 | IN-Sketch | 1000 | 389 | |||||
| MetaCLIP-B16 | IN-V2 | 1000 | 405 | |||||
| MetaCLIP-B16 | StanfordCars | 196 | 279 | |||||
| MetaCLIP-B16 | Food101 | 101 | 247 |
Test-Time Adaptation of VLMs
Test-time adaptation of vision-language models uses unlabeled deployment data to improve predictions under domain shifts without retraining.
See all 91 papers